Empirical Formal Methods: Guidelines for Performing Empirical Studies on Formal Methods

Empirical studies on formal methods and tools are rare. In this paper, we provide guidelines for such studies. We mention their main ingredients then define nine different study strategies (usability testing, laboratory experiments with software human subjects, case studies, qualitative surveys, judgement systematic literature reviews, mapping studies) discuss each of them crucial characteristics, the difficulties applying to tools, typical threats validity, maturity in methods, pointers external guidelines, other fields. conclude a number challenges empirical methods.